{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines (SVMs)\n",
    "\n",
    "Support Vector Machines (SVMs) are a type of machine learning algorithm used for classification and regression analysis. In particular, linear SVMs are used for binary classification problems where the goal is to separate two classes by a hyperplane.\n",
    "\n",
    "The hyperplane is a line that divides the feature space into two regions. The SVM algorithm tries to find the hyperplane that maximizes the margin, which is the distance between the hyperplane and the closest points from each class. The points closest to the hyperplane are called support vectors and play a crucial role in the algorithm's optimization process.\n",
    "\n",
    "In linear SVMs, the hyperplane is defined by a linear function of the input features. The algorithm tries to find the optimal values of the coefficients of this function, called weights, that maximize the margin. This optimization problem can be formulated as a quadratic programming problem, which can be efficiently solved using standard optimization techniques.\n",
    "\n",
    "In addition to finding the optimal hyperplane, SVMs can also handle non-linearly separable data by using a kernel trick. This technique maps the input features into a higher-dimensional space, where they might become linearly separable. The SVM algorithm then finds the optimal hyperplane in this transformed feature space, which corresponds to a non-linear decision boundary in the original feature space.\n",
    "\n",
    "Linear SVMs have been widely used in many applications, including text classification, image classification, and bioinformatics. They have the advantage of being computationally efficient and easy to interpret. However, they may not perform well in highly non-linearly separable datasets, where non-linear SVMs may be a better choice."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class SVM:\n",
    "    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iters=1000):\n",
    "        self.lr = learning_rate\n",
    "        self.lambda_param = lambda_param\n",
    "        self.n_iters = n_iters\n",
    "        self.w = None\n",
    "        self.b = None\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        n_samples, n_features = X.shape\n",
    "        y_ = np.where(y <= 0, -1, 1)\n",
    "        self.w = np.zeros(n_features)\n",
    "        self.b = 0\n",
    "\n",
    "        # Gradient descent\n",
    "        for _ in range(self.n_iters):\n",
    "            for idx, x_i in enumerate(X):\n",
    "                condition = y_[idx] * (np.dot(x_i, self.w) - self.b) >= 1\n",
    "                if condition:\n",
    "                    self.w -= self.lr * (2 * self.lambda_param * self.w)\n",
    "                else:\n",
    "                    self.w -= self.lr * (2 * self.lambda_param * self.w - np.dot(x_i, y_[idx]))\n",
    "                    self.b -= self.lr * y_[idx]\n",
    "\n",
    "    def predict(self, X):\n",
    "        linear_output = np.dot(X, self.w) - self.b\n",
    "        return np.sign(linear_output)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "# Example usage\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X, y = datasets.make_blobs(n_samples=100, centers=2, random_state=42)\n",
    "y = np.where(y == 0, -1, 1)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "svm = SVM()\n",
    "svm.fit(X_train, y_train)\n",
    "y_pred = svm.predict(X_test)\n",
    "\n",
    "\n",
    "# Evaluate model\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy:\", accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.5\n"
     ]
    }
   ],
   "source": [
    "# Generate data\n",
    "X, y = make_classification(n_features=5, n_samples=100, n_informative=5, n_redundant=0, n_classes=2, random_state=1)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)\n",
    "\n",
    "# Initialize SVM model\n",
    "svm = SVM()\n",
    "\n",
    "# Train model\n",
    "svm.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = svm.predict(X_test)\n",
    "\n",
    "# Evaluate model\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy:\", accuracy)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
